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1 Introduction 

The  NIST  Algorithm  Testing  and  Evaluation  Program  for  Coordinate  Measuring  Systems 
(ATEP-CMS)  evaluates  the  performance  of  data  analysis  software  used  in  coordinate  measuring 
systems.  ATEP-CMS  is  available  through  the  NIST  Calibration  Program  as  Special  Test  numbers 
10070S  (special  test  of  CMS  software  with  NIST-generated  data  sets)  and  10080S  (special  test  of 
CMS  software  with  customer-generated  data  sets).  ATEP-CMS  supports  the  following  geometry 
types;  circle,  line,  plane,  sphere,  cylinder,  cone,  and  torus. 

This  report  details  the  CMS  testing  and  evaluating  methods  that  ATEP-CMS  uses  to  test  and 
evaluate  the  performance  of  CMS  software.  It  is  intended  as  a guide  to  procedures  for  ATEP-CMS 
testing  personnel. 

Section  2 of  this  report  discusses  the  specific  test  methods  for  carrying  out  an  ATEP-CMS 
request.  Section  3 describes  the  general  principles,  methods,  and  practices  for  testing  the 
performance  of  CMS  software.  This  report  also  contains  appendices  that  provide  information  on  the 
characteristics  by  which  CMS  software  performance  is  evaluated,  a template  of  a special  test  report, 
and  other  pertinent  information. 

2 Specific  Test  Procedures 

This  section  describes  the  specific  steps  that  must  be  carried  out  in  order  to  test  a CMS 
software  package.  The  emphasis  in  this  section  is  on  the  administrative  and  logistical  aspects  of  the 
test.  Technical  aspects  of  the  activities  are  discussed  in  Section  3.  This  section  provides  cross- 
references  to  Section  3 and  references  to  other  documents  as  needed. 

ATEP-CMS  customer  requests  are  received  through  the  Manufacturing  Systems  Integration 
Division  (MSID)  of  NIST.  The  customer  and  MSID  discuss  the  details  of  the  customer’s  testing 
needs  and  MSID’s  testing  options.  The  customer  and  MSID  agree  on  the  ranges  of  parameters 
(geometry  types,  form  errors,  measurement  errors,  and  sampling  plans)  to  be  used  in  defining  the 
customer's  test  experiments  (details  provided  in  Section  3.2).  The  customer  sends  MSID  a purchase 
order  requesting  ATEP-CMS  service.  MSID  directs  the  purchase  order  to  the  Calibration  Program 
(CP)  for  handling.  CP  then  provides  MSID  with  a formal  test  folder.  Following  are  the  phases  and 
detailed  descriptions  of  the  steps  to  follow  in  completing  a customer's  request  for  ATEP-CMS 
service.  ATEP-CMS  testing  personnel  carry  out  the  instructions  printed  in  italics.  The  descriptions 
following  the  italicized  instructions  provide  details  and  cross-references  to  Section  3 where 
appropriate. 
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Manufacturing  Systems  Integration  Division 


Phase  I:  Handling  the  test  service  request 

1.  Log  receipt  of  the  test  folder  for  each  ATEP-CMS  request  MSID  receives  from  CP.  The 
test  folder  includes  a serially  numbered  test  form  (form  number  NIST  64)  and  the 
customer's  purchase  order.  The  test  form  has  six  parts,  to  allow  appropriate  record 
keeping  at  each  office.  These  six  parts  are  as  follows: 

• 1-  acceptance  form; 

• 2-  division  record  form; 

• 3-  fee  record  form; 

• 4-  shipping  record  form; 

• 5-  packing  slip  form; 

• 6-  CP  record  form. 

MSID  receives  copies  #1  - #5  of  the  test  form.  CP  retains  copy  #6  for  their  records. 
Please  note  that  all  technical,  administrative,  and  scheduling  aspects  of  the  test  should  be 
recorded  throughout  the  testing  process. 

2.  Send  the  acceptance  form  copy  of  the  test  form  to  the  customer.  This  form  includes  the 
estimated  cost  for  the  service  requested,  an  estimated  date  for  the  customer  to  expect 
receipt  of  the  data  sets,  and  an  estimated  date  for  the  completion  of  the  test.  The  fee 
schedule  can  be  obtained  from  the  CP  office. 

Phase  II:  Setting  up  the  test 

3.  Design  specific  values  for  test  description  definitions  and  number  of  tests  for  each 
geometry  ordered.  Any  form  errors,  sampling  plans,  and  measurement  errors  are  defined 
at  this  time.  For  technical  details  refer  to  Section  3.2:  Designing  test  descriptions. 

4.  Create  experiment  and  enter  test  definitions  in  the  NIST  Algorithm  Testing  System  (ATS) 
[3,4].  For  technical  details  refer  to  Section  3.3.1:  Entering  test  definitions,  steps  1-4. 

5.  Generate  data  sets  from  test  definitions.  (Fit  results  can  be  generated  at  this  time  and 
saved  for  when  customer's  fit  results  are  returned).  For  technical  details  see  Section  3.3.1: 
Entering  test  definitions,  step  5. 

6.  Export  data  sets  to  file  formats  defined  by  customer.  Keep  backup  copies  of  all  data  sets. 
For  details  see  Section  3.3.2:  Exporting  and  Mailing  Data  Sets. 

7.  Mail  data  sets  to  customer  with  instructions.  Indieate  format  requirements  for  fit  results. 
Include  copy  #4  of  the  test  form  with  the  diskette.  For  details  see  Seetion  3.3.2:  Exporting 
and  Mailing  Data  Sets. 

Phase  III:  Receiving  fit  results 

8.  Log  receipt  of  diskette  with  customer’s  fit  results. 
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9.  Send  acknowledgment  of  receipt  of  diskette  with  fit  results  to  customer. 

10.  Import  customer's  fit  results  into  the  ATS  under  the  customer's  previously  defined 
experiment.  If  there  are  any  problems,  contact  the  customer  to  resolve  them.  For  details 
see  Section  3.3.3:  Importing  Customer’s  Data. 

Phase  rV:  Testing  fit  results 

11.  Generate  reference  fits  with  the  ATS.  If  reference  fits  were  generated  during  step  5 above, 
then  this  step  is  skipped.  For  technical  details  see  the  ATS’s  users  guide,  the  reference 
manual  [3,4],  and  the  ATS  on-line  help. 

12.  Run  the  necessary  analyses.  For  technical  details  see  Section  3.3.4:  Run  Analyses  and 
Prepare  Special  Test  Report  and  the  ATS  on-line  help. 

13.  Save  results  and  secure  files. 

14.  Note  discrepancies,  resolve  with  customer  and  repeat  tests  and  analyses  as  necessary. 

Phase  V:  Preparing  Special  Test  Report 

15.  Prepare  Report  of  Special  Test  Service.  Refer  to  Section  3.3.4:  Run  Analyses  and 
Prepare  Special  Test  Report.  Also  see  Appendix  C for  a template  of  the  Special  Test 
Report. 

16.  Secure  Report  of  Special  Test  Service  and  all  related  fit  result  files. 

17.  Obtain  appropriate  signatures  (if  required,  OU  director  and  division  chief).  Minimally, 
the  calibration  supervisor  needs  to  sign  the  report. 

18.  Send  the  diskette  with  the  results  and  the  printed  Special  Test  Report  to  the  customer. 
Include  copy  #5  of  the  test  form  in  the  package  sent  to  the  customer. 

Phase  VI:  Requesting  payment 

19.  Indicate  on  the  fee  record  form  of  the  test  form  (copy  #3)  the  fee  to  be  charged  along 
with  the  cost  center. 

20.  Retain  division  form  copy  of  the  test  form  (copy  #2)  for  record  within  the  division. 

21.  Forward  the  test  form  with  completed  work  to  CP  for  filing  (copy  #3  of  the  test  form). 
CP  audits  fee  record  (test  form  copy  #3)  and  sends  it  to  the  Office  of  the  Comptroller  for 
billing.  After  billing.  Office  of  the  Comptroller  returns  the  fee  record  to  CP  for  filing. 

22.  Log  completion  of  service  to  customer.  Record  technical,  administrative,  and  scheduling 
aspects  of  the  test. 
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3 Test  Methodology 

This  section  describes  the  test  methodology  used  in  ATEP-CMS.  This  section  provides  the 
underlying  technical  details  for  Section  2,  a description  of  the  general  principles,  the  methods,  and 
the  practices  for  testing  the  performance  of  CMS  software. 

ATEP-CMS  evaluates  CMS  software  strictly  in  terms  of  its  intended  function.  The  software 
is  considered  to  be  a "black  box"  that  is  subjected  to  variations  of  inputs  while  outputs  are  evaluated 
with  respect  to  a specific  objective.  This  functional  testing  determines  how  well  the  software  does 
what  it  is  supposed  to  do.  This  is  done  by  designing  data  sets  which  reflect  the  expected  range  and 
variability  of  actual  data.  The  NIST  Algorithm  Testing  System  (ATS)  is  the  tool  used  to  perform 
the  functional  testing. 

Data  formats,  data  resolution,  and  related  characteristics  of  the  test  are  subject  to  agreement 
between  the  customer  and  NIST. 

3.1  The  Testing  Tool:  Algorithm  Testing  System  (ATS) 

3.1.1  Overview 

The  ATS  consists  of  three  major  components:  a data  generator,  a set  of  reference  algorithms, 
and  a fit  analysis  module.  The  data  generator  produces  data  sets  based  on  a test  description  provided 
by  the  ATS  operator.  A test  description  consists  of  ranges  of  values  for  each  of  four  information 
classes: 


1.  nominal  (ideal)  geometry  of  the  feature; 

2.  form  errors  of  the  feature  being  simulated; 

3.  sampling  plan  (number  and  location  of  points  on  the  feature); 

4.  measurement  errors  statistical  distribution  of  the  points. 

The  data  sets  are  processed  by  the  software  under  test  to  generate  test  fits  and  by  ATS 
algorithms  to  generate  reference  fits.  The  fits  are  analyzed  in  terms  of  geometric  differences  among 
the  fits.  Test  fits  and  reference  fits  are  compared  to  evaluate  the  geometric  differences  between 
them.  The  data  sets,  test  fits,  and  reference  fits  are  used  to  assess  the  performance  of  the  software 
under  test. 

The  ATS  features  the  following  implementations: 

• least-squares  fitting  algorithms; 

• 3-D  Least-squares  for  circles; 

• minimum-radius  circumscribed  for  circles  and  spheres; 

• data  generation  with  simulated  form  errors,  measurement  errors,  and  sampling  plans; 

• window-based  user  interface; 

• on-line  help; 

• data  management  capabilities; 

• residual  analysis  of  a fit; 

• detailed  comparison  of  fit  results; 
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• graphical  display  of  data  sets  and  fit  results; 

• printing  capabilities; 

• detailed  report  of  results. 

For  specific  technical  details  regarding  how  to  run  the  ATS  and  how  to  create  an  experiment 
for  CMS  software  performance  testing,  please  refer  to  the  Reference  Manual  for  the  Algorithm 
Testing  System  Version  2 [3],  and  the  User’s  Guide  for  the  Algorithm  Testing  System  Version  2 [4]. 

3.1.2  Functions  of  the  ATS 

The  following  subsections  briefly  describe  the  basic  functions  of  the  ATS. 

3.1.2.1  Test  Definition 

Test  definitions  describe  the  geometric  features  to  be  fitted.  The  test  definition  contains 
the  nominal  geometry,  the  form  errors,  the  sampling  plan,  and  the  measurement  errors  to  be 
simulated.  In  the  ATS,  the  test  definition  is  defined  after  starting  an  experiment  and 
selecting  a geometry  type.  Once  the  new  test  definition  is  inserted,  the  user  interface  for 
describing  the  test  is  provided.  Data  is  entered  for  each  of  the  information  classes. 

3.1.2.2  Data  Generator 

The  data  generator  is  used  to  generate  data  sets  for  specified  feature  types  with  controlled 
range  and  variability  as  well  as  the  known  solution  for  some  types  of  features.  The  data 
generator  uses  the  test  description  as  input  to  generate  the  test  data. 

3. 1.2.3  Internal  Algorithms 

The  ATS  contains  internal  fitting  algorithms.  These  internal  algorithms  are  used  to 
generate  reference  fits  corresponding  to  specific  data  sets.  Currently,  the  ATS  implements 
least-squares  for  all  geometric  elements,  3-D  least  squares  for  circles,  and  minimum 
circumscribed  algorithms  for  circles  and  spheres. 

3.1.2.4  Data  Import  and  Export 

The  ATS  incorporates  means  to  transfer  data  sets  to  the  software  under  test,  and  to 
receive  fit  results  from  the  software  under  test.  The  formats  required  are  detailed  in 
Appendix  B of  the  draft  national  standard,  ASME  B89.4.10  [1],  and  in  the  ATS  reference 
manual  [3]. 

3.1.2.5  Fit  Analysis 

The  fit  analysis  module  is  used  for  comparing  the  results  of  the  software  under  test  to  the 
reference  results  with  respect  to  the  objective  function.  The  comparison  is  based  on  the 
geometric  differences  between  the  test  fits  and  the  reference  fits. 

3.2  Designing  test  descriptions 

In  order  to  design  test  descriptions  for  a particular  test  request  NIST  testing  personnel  needs 
to  have  some  knowledge  of  the  typical  CMS  inspection  scenarios  that  the  customer  works  with.  This 
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section  provides  a checklist  of  the  information  that  NIST  testing  personnel  should  ask  the  customer 
for  before  designing  a test  description.  Some  examples  of  the  type  of  information  customers  might 
provide  are  listed  within  the  checklist.  Several  of  these  items  may  have  different  answers  for  each 
geometry  type. 

□ What  is  the  number  of  data  sets  typically  used? 

EXAMPLE:  20  data  sets,  100  data  sets,  2000  data  sets,  etc. 

□ What  unit  of  measurement  is  used? 

EXAMPLE:  inches,  millimeters,  etc. 

□ What  geometric  element  types  are  of  interest? 

EXAMPLE:  line,  circle,  plane,  sphere,  cone,  cylinder,  torus. 

□ What  is  the  range  of  part  sizes  typically  seen? 

EXAMPLE:  Customer  has  small  and  large  parts  that  fill  most  of  the  space  in  an 
inspection  tray.  Small  parts  are  about  20  or  30  mm.  Large  parts  are  about  V2  the 
working  volume  or  more  in  each  dimension. 

□ Where  should  the  part  origin  be,  at  the  center  of  the  part  or  at  a distance  away  from  the 
origin? 

EXAMPLE:  Origin  is  typically  10  meters  away  from  the  part  origin. 

□ How  many  significant  digits  are  to  be  considered  in  the  test? 

EXAMPLE:  Resolution  of  10‘^  meter 

□ How  many  data  sets  per  geometry  should  be  generated? 

EXAMPLE:  50  data  sets  per  geometry  type 

□ What  kind  of  form  errors  are  typically  seen  on  parts?  (This  may  vary  with  the  geometry 
types). 

EXAMPLE:  random,  sine,  step,  bends,  etc. 

□ What  sampling  strategy  is  used?  How  many  points  are  taken  with  each  measurement? 
(This  will  vary  with  the  geometry  type). 

EXAMPLE:  random,  equispaced,  stratified 

□ What  measurement  errors  are  typically  seen? 

EXAMPLE:  uniform,  normal 

□ If  testing  circles,  are  all  the  circles  in  the  x-y  plane  or  are  the  circles  skewed? 

□ Are  there  partial  features  for  each  kind  of  geometry  type  to  be  tested? 
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EXAMPLE:  an  arc  of  a circle 


In  general,  circle  geometry  types  are  difficult  to  test  because  the  circle  could  be  skewed  or 
not.  Planes,  spheres,  and  tori  are  generally  simple  to  test.  Cones  and  cylinders  are  complicated  by 
the  large  range  of  form  errors  possible.  The  ATS  reference  manual  and  user  guide  [3,4]  provide  the 
technical  detail  necessary  to  design  geometric  features  with  form  errors,  measurement  errors,  and 
sampling  strategies. 

3.3  Steps  for  CMS  software  performance  evaluation 
3.3.1  Entering  Test  Deflnitions 

The  data  sets  to  be  used  to  generate  reference  fit  results  are  generated  by  either  the  ATS  or 
the  customer.  If  the  data  sets  are  customer-generated  there  is  no  need  to  enter  a test  definition.  (Test 
definitions  are  necessary  for  generating  data  sets  in  the  ATS).  The  customer-generated  data  sets 
must  be  verified  for  appropriate  format  and  then  imported  into  the  ATS.  For  data  set  file  formats 
please  refer  to  Appendix  B of  the  national  draft  standard,  ASME  B89.4. 10  [1]  and  the  ATS  reference 
manual  [3].  Also  note  that  the  ATS  provides  on-line  help  for  all  of  its  functions. 

If  the  data  sets  are  to  be  generated  with  the  ATS,  the  following  five  steps  must  be  followed. 
The  actual  data  entered  into  the  ATS  depends  on  the  information  gathered  from  the  checklist 
prepared  in  section  3.2. 

Step  1.  Run  the  ATS  and  define  the  nominal  geometry  for  the  specified  feature.  The 
parameters  will  vary  depending  on  the  geometry.  The  following  table  provides  a general 
guideline  of  the  parameters  required  for  each  geometry.  Use  this  table  as  a reference  when 
going  through  the  checklist  in  section  3.2  with  the  customer. 


line 

circle 

planes 

sphere 

cylinder 

cone 

torus 

location 

X 

X 

X 

X 

X 

X 

X 

direction 

X 

X 

X 

X 

X 

X 

size 

X 

X 

X 

X 

X 

angle 

X 

extent 

X 

X 

X 

X 

X 

X 

X 

Step  2.  Define  form  errors  to  be  simulated.  Form  errors  simulate  part  form  errors  that  the 
customer  typically  sees  in  their  manufactured  parts.  The  ATS  simulates  random,  sinusoidal, 
step,  and  bend  errors.  These  form  errors  are  also  specific  to  the  geometry  type.  Use  the 
following  two  tables  as  guidelines  when  going  though  the  checklist  in  section  3.2  with  the 
customer  and  when  designing  the  test  description. 
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random 

sine 

step 

bend 

line 

X 

X 

X 

X 

circle 

X 

X 

X 

plane 

X 

X 

X 

X 

sphere 

X 

X 

torus 

X 

X 

random 

axis 

sine 

axis 

bend 

axis 

step 

surface 

sine 

radius 

taper 

along 

axis 

radius 

step 

along 

axis 

radius 

step 

around 

axis 

cylinder 

X 

X 

X 

X 

X 

X 

X 

X 

cone 

X 

X 

X 

X 

X 

X 

X 

X 

Step  3.  Identify  the  sampling  plan  (acquire  from  the  customer)  to  be  simulated.  The  ATS 
simulates  the  following  sampling  methods. 

• random 

• equispaced 

• stratified 

Random  sampling  means  that  each  data  point  is  selected  at  random  from  within  the  extent 
of  the  geometry.  For  circles  and  lines,  a uniformly-distributed  random  number  is  generated. 
For  surfaces,  two  random  numbers  are  generated. 

Equispaced  sampling  means  that  the  surface  is  divided  into  a grid  by  dividing  the  extent 
into  bins  according  to  the  number  of  points  indicated.  One  point  is  selected  at  the  center  of 
each  grid  box. 

Stratified  sampling  is  similar  to  equispaced  sampling.  Rather  than  sampling  at  the  center 
of  each  box,  however,  each  sample  point  is  obtained  by  generating  a uniformly-distributed 
random  number  (or  two  numbers,  for  surfaces)  between  the  limits  of  the  grid  box. 

Step  4.  Identify  the  measurement  errors  (acquire  from  the  customer)  to  be  simulated,  if  any. 
The  ATS  simulates  the  following  types  of  measurement  errors. 

• uniform 

• normal 
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Measurement  error  is  a displacement  of  a sampled  point  in  a random  direction  in  three 
dimensions.  The  ATS  adds  a measurement  error  to  each  sample  point  after  it  is  generated. 
The  measurement  error  at  a point  is  independent  of  the  error  at  any  other  point,  anc  s also 
independent  of  the  geometry.  All  measurement  errors  arc  random.  You  can  control  whether 
measurement  errors  are  to  be  added,  and  you  can  define  a scale  for  the  distribution  of  the 
measurement  error. 

Step  5.  Generate  data  sets.  The  ATS  provides  a function  for  generating  the  data  sets.  The 
ATS  also  provides  the  option  to  generate  fit  results  while  generating  the  data  sets.  Filename 
defaults  are  provided  for  the  data  sets  and  the  fits  but  can  be  overridden  by  the  ATS  user. 

Generated  test  data  sets  will  be  typical  of  data  normally  acquired  by  the  CMS.  The  data 
resolution  must  be  considered  when  generating  data  sets.  The  resolution  of  input  data  should 
match  the  resolution  of  the  CMS  scale  systems.  Testing  personnel  should  make  sure  the 
customer  has  described  in  detail  the  CMS  scale  systems  typically  used  by  the  customer. 

3.3.2  Exporting  and  Mailing  Data  Sets 

ATS-generated  data  sets  prepared  for  a specific  customer  experiment  are  exported  from  the 
ATS  to  files  of  ASCII  text  representations  of  the  data.  The  ATS  provides  a function  for  exporting 
the  data  sets.  The  ATS  also  provides  a function  to  save  generated  data  sets.  Save  backups  of  all  data 
sets  generated  by  the  ATS.  The  ATS  generates  a separate  file  for  each  data  set.  Each  data  set  file 
produced  by  the  ATS  contains  a data  set  identification  name,  a geometry  type  label,  the  date  the  data 
set  was  generated,  the  number  of  points  represented  in  the  file,  and  the  coordinates  of  the  data.  For 
further  details  and  examples  please  see  Section  B.3.1  of  Appendix  B in  the  ASME  B89.4.10  draft 
national  standard,  B89.4.10  [1]  and  the  ATS  reference  manual  [3].  If  the  customer  defined  file 
formats  for  receiving  the  data  sets,  export  the  data  sets  to  the  specified  format. 

Once  the  data  sets  are  exported,  MSID  mails  the  customer  a diskette  contain  the  data  set  files 
and  an  instruction  sheet  for  the  customer.  Please  refer  to  Appendix  A for  a copy  of  the  instruction 
sheet. 

3.3.3  Importing  Customer’s  Data 

After  the  customer  runs  the  test  software  using  the  ATS-generated  data  sets  the  fit  results  are 
shipped  back  to  NIST  to  complete  evaluation  of  the  customer’s  CMS  software.  Fit  results  are 
verified  to  assure  appropriate  format.  Details  on  the  appropriate  file  formats  are  found  in  the  ASME 
B89.4.10  draft  national  standard  [1].  The  ATS  provides  a function  for  importing  data  sets  and  fit 
results.  Refer  to  the  ATS  reference  manual  [3]  for  further  details. 

For  customer-generated  data  sets,  customers  also  provide  NIST  with  corresponding  fit 
results.  Both  the  data  sets  and  the  fits  are  imported  to  the  ATS.  NIST  then  generates  reference  fit 
results  for  the  data  sets  provided  and  compares  the  reference  fits  to  the  customer’s  fits. 

3.3.4  Run  Analyses  and  Prepare  Special  Test  Report 

The  customer’s  test  fit  results  are  compared  to  the  ATS’s  reference  results.  The  parameters 
compared  vary  with  the  type  of  geometry.  For  more  details  on  the  parameters  compared  for  each 
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type  of  geometry  refer  to  Appendix  B of  this  document. 

The  comparison  is  the  geometric  difference  between  the  test  fits  and  the  reference  fits.  The 
computations  for  the  geometric  difference  depend  on  the  geometry  type.  The  guidelines  for 
algorithm  comparison  in  Section  4.1  and  the  specifications  for  reporting  test  results  in  Section  5.5 
of  the  ASME  B89.4.10  draft  national  standard  [1]  are  used  to  compare  the  results  of  the  software 
under  test  to  the  reference  results  and  to  prepare  the  Special  Test  Report. 

The  ATS  provides  an  analysis  as  a result  of  the  comparison.  For  technical  details  refer  to  the 
ATS  reference  manual  [3],  the  ATS  online  help,  and  the  report  Performance  Measures  for 
Geometric  Fitting  in  the  NIST  Algorithm  Testing  and  Evaluation  Program  [2]. 

The  Special  Test  Report  is  prepared  after  the  comparisons  are  completed.  The  Special  Test 
Report  includes  the  following: 

• the  geometric  feature  types  tested; 

• the  range  of  conditions  represented  by  the  test  data; 

• characteristics  of  the  software  under  test 

computing  environment,  software  version,  etc.; 

• the  root-mean-square  deviations  between  the  software  under  test  and  the  reference  results; 

• uncertainty  of  the  observed  average  deviations. 

For  more  details  see  the  Special  Test  Report  template  in  Appendix  C. 
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Appendix  A: 

Customer 
Instruction  Sheet 


Template 


Customer  Instruction  Sheet 

The  following  instructions  are  intended  for  the  customer  personnel  who  will  be  running  the 
CMS  software  to  be  tested.  A DOS  formatted  diskette  containing  files  with  NIST-generated  data 
sets  is  enclosed.  The  files  contain  data  sets  of  3-dimensional  coordinate  values  generated  by  the 
NIST  Algorithm  Testing  System  (ATS).  All  file  formats  are  ASCII  text  representations  of  the  data. 
All  instructions  for  the  customer  personnel  are  listed  in  italics.  The  descriptions  following  the 
italicized  instructions  provide  supporting  details.  Please  follow  all  instructions  carefully. 

The  NIST-generated  data  sets  are  intended  to  be  used  as  input  data  to  the  CMS  software  to 
be  tested.  The  CMS  software  under  test  produces  fit  results  corresponding  to  the  NIST-generated 
data  sets. 

Step  1 . Copy  the  NIST- generated  data  sets  from  the  enclosed  diskette  to  your  system.  Note  that  more 
than  one  diskette  may  be  included.  The  diskette  contains  a separate  file  for  each  data  set.  The  data 
set  files  have  the  following  format: 

Line  1:  ID  GEOMETRY_TYPE  DATE 

Line  2:  COUNT 

Line  3:  COORDINATES 

Line  4:  COORDINATES 

In  Line  1 , ID  is  an  identifier  string  of  no  more  than  32  characters  that  uniquely  identifies  the  data  set. 
GEOMETRY_TYPE  is  a label  of  the  geometry  type  represented  by  the  data.  The  geometry  types 
currently  supported  in  ATEP-CMS  are  circles,  cones,  cylinders,  lines,  planes,  spheres,  and  tori. 
DATE  is  the  date  on  which  the  data  file  was  generated,  represented  in  the  format:  dd-mmm-yyyy 
(e.g.,  26- APR-1995). 


On  Line  2,  COUNT  is  an  integer  count  of  the  number  of  points  represented  in  the  rest  of  the  file. 
Starting  on  Line  3,  each  remaining  line  represents  one  point  of  the  data  set,  using  three  Cartesian 
coordinate  values  in  order  X,  Y,  and  Z.  The  coordinate  values  are  dimensionless,  but  are  all  to  the 
same  scale.  Coordinate  values  are  represented  in  decimal  floating  point  notation,  separated  by 
spaces.  The  coordinate  values  are  represented  with  as  many  digits  as  necessary  to  represent  the 
values  to  the  resolution  used  in  generating  the  data.  Each  line  is  terminated  by  an  end-of-line 
sequence  (ASCII  codes  13  and  10)  following  the  Z coordinate  value.  After  the  end-of-line  sequence 
for  the  last  point,  the  file  ends. 

Step  2.  Convert  the  NIST- generated  data  set  files  to  the  format  required  by  the  CMS  software  under 
test.  This  step  is  unique  to  each  individual  CMS  software  used  by  the  customer. 

Step  3.  Download  the  data  sets  to  the  CMS  software  under  test.  This  step  is  unique  to  each 
individual  CMS  software  used  by  the  customer. 

Step  4.  Run  the  CMS  software  under  test  and  produce  fit  results  for  each  data  set.  This  step  is 
unique  to  each  individual  CMS  software  used  by  the  customer. 

Step  5.  Convert  the  fit  results  to  the  file  format  specified  by  NIST.  Fit  results  are  returned  to  NIST 
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in  files  formatted  according  to  the  geometry  type.  Each  file  represents  a single  fit  result  for  a single 
data  set.  The  first  line  of  the  fit  result  file  should  be  a copy  of  the  first  line  of  the  data  set  file  from 
which  the  fit  was  calculated.  The  remainder  of  the  file  contains  the  parameters  of  the  fit. 

The  parameters  and  their  expected  sequence  in  the  file  are  described  below.  Parameter 
values  must  be  separated  by  white  space.  (White  space  consists  of  one  or  more  of  the  ASCII  codes 
for  space,  tab,  carriage  return,  line  feed,  or  form  feed.)  Any  line  after  the  first  may  contain 
comments.  A comment  starts  with  a semicolon  (;)  and  continues  to  the  end  of  the  line.  Comments 
are  treated  like  white  space.  All  position  and  size  parameters  are  dimensionless,  but  must  be 
reported  in  the  same  scale  as  the  data  set  coordinates.  All  angle  values  must  be  in  decimal  degrees. 
Parameters  may  appear  in  decimal  floating  point  or  exponential  (i.e.,  ±x.xxe±xx)  notation.  The 
precision  of  the  parameters  is  assumed  to  be  exact;  that  is,  the  values  are  assumed  to  include  trailing 
zeroes  to  infinity.^ 

The  fit  parameters  expected  in  result  files  correspond  to  the  parameterization  used  in  the  ATS 
to  calculate  the  difference  between  fits.  These  parameters  and  their  expected  sequence  in  the  file  are 


Parameter 

Line 

Plane 

Circle 

Sphere 

Cylinder 

Cone 

Torus 

position 

X 

1 

1 

1 

1 

1 

1 

1 

Y 

2 

2 

2 

2 

2 

2 

2 

Z 

3 

3 

3 

3 

3 

3 

3 

orientation 

X 

4 

4 

4 

4 

4 

4 

Y 

5 

5 

5 

5 

5 

5 

Z 

6 

6 

6 

6 

6 

6 

diameter 

major 

7 

4 

7 

7 

minor 

8 

position 

7 

half-angle 

8 

NOTES:  (1)  The  position  parameter  for  lines  is  any  point  on  the  line;  for  cylinders  it  is  any  point  on  the 

axis;  for  planes  it  is  any  point  on  the  plane. 

(2)  The  orientation  parameters  must  be  proportional  to  the  direction  cosines  for  the  indicated 
direction.  For  circles,  planes,  and  tori,  the  orientation  is  that  of  the  normal  to  the  plane  of  the 
fit. 

(3)  The  position  parameter  for  a cone  is  the  perpendicular  distance  from  the  midpoint  of  the 
bounded  axis  to  the  surface  of  the  cone. 


^ Since  fit  comparisons  involve  calculation  of  geometric  differences,  rather  than  direct 
comparison  of  parameter  values,  fit  parameters  should  be  expressed  in  as  much  precision  as 
possible.  In  particular,  it  is  important  to  not  round  the  fit  parameters  to  the  apparent  resolution  of 
the  data  set. 
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Step  6.  Copy  fit  result  files  onto  diskettes  and  ship  to  NIST.  If  the  files  are  shipped  to  NIST  in  a 
format  other  than  the  specified  by  NIST,  please  provide  a file  format  description.  Following  is  the 
shipping  address  to  be  used: 

ATEP-CMS  Special  Test  Service 
National  Institute  of  Standards  and  Technology 
Building  220,  Room  A 127 
Gaithersburg,  MD  20899-0001 

If  at  any  time  there  are  questions  or  problems,  please  contact  the  NIST  Manufacturing  Systems 
Integration  Division.  Contact  people  are: 


Cathleen  Diaz 

(301)975-2889 

diaz@cme.nist.gov 


Ted  Hopp 

(301)975-3545 

hopp@cme.nist.gov 
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Appendix  B: 

Software  Performance 
Characteristics 


Appendix  B:  Software  Performance  Characteristics 

ATEP-CMS  is  concerned  with  the  performance  characteristics  that  affect  the  uncertainty  of 
measurement  results  produced  by  CMSs.  Quality  and  reliability  are  two  of  these  software 
performance  characteristics.  For  more  technical  details  on  this  section  please  refer  to  the  national 
draft  standard,  ASME  B89.4.10  [1]. 

Evaluation  of  Quality 

ATEP-CMS  measures  the  performance  of  geometric  fitting  software  used  in  CMSs. 
Geometric  fitting  is  the  process  of  computing  the  representation  parameters  of  a geometric  element 
that  in  some  sense  best  represents  a set  of  point  coordinate  data  (substitute  geometry).  ATEP-CMS 
provides  quantitative  measures  of  performance.  This  measurement  provides  inspection  planners  with 
the  quantitative  measures  of  performance  needed  to  develop  uncertainty  budgets  and  to  evaluate  the 
quality  of  measurement  results. 

In  summary,  the  performance  measures  used  in  ATEP-CMS  quantify  how  well  the  software 
computes  substitute  geometries  over  a range  of  inspection  problems  [2].  The  goal  is  to  provide  to 
a user  of  the  software  information  necessary  to  decide  whether  the  software  is  adequate  for  a 
particular  application.  As  a result,  the  following  criteria  in  developing  ATEP-CMS  performance 
measures  are  used: 

• each  measure  should  be  directly  related  to  inspection  tasks; 

• each  measure  should  combine  like  a standard  deviation  with  other  sources  of  uncertainty 
in  a CMS; 

• each  measure  should  have  reasonable  probability  and  coverage  interpretations; 

• an  estimate  of  uncertainty  should  be  derivable  for  each  measure. 

Evaluation  Parameters 

Each  geometry  type  has  a unique  set  of  evaluation  parameters.  Test  results  are  reported  as 
follows. 

LINE 

a.  the  largest  separation  distance  from  the  bounded  test  line  to  the  reference  line 

b.  the  angle  between  the  test  and  reference  features 

CIRCLE 

a.  the  absolute  difference  between  the  radii  of  the  test  and  reference  circles 

b.  the  distance  between  the  centers  of  the  test  and  reference  circles 

c.  the  angle  between  the  planes  of  the  test  and  reference  circles  if  applicable 

PLANE 

a.  the  largest  separation  distance  from  the  bounded  test  plane  to  the  reference  plane 

b.  the  angle  between  the  test  and  reference  planes 
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SPHERE 

a.  the  absolute  difference  between  the  radii  of  the  test  and  reference  spheres 

b.  the  distance  between  the  centers  of  the  test  and  reference  spheres 

CYLINDER 

a.  the  absolute  difference  between  the  radii  of  the  test  and  reference  cylinders 

b.  the  maximum  distance  from  the  bounded  test  cylinder  axis  to  the  axis  of  the  reference 
cylinder 

c.  the  angle  between  the  axes  of  the  test  and  reference  cylinders 
CONE 

a.  For  each  cone,  the  perpendicular  distance  from  the  midpoint  of  the  bounded  axis  to  the 
corresponding  cone  surface  is  computed.  The  evaluation  parameter  is  the  absolute  difference 
between  these  distances. 

b.  the  maximum  distance  from  the  bounded  test  axis  to  the  reference  axis 

c.  the  angle  between  the  test  and  reference  axes 

d.  the  absolute  difference  between  the  test  and  reference  included  cone  angles 
TORUS 

a.  the  absolute  difference  between  the  major  radii  of  the  test  and  reference  torus  and  the  minor 
radii  of  the  test  and  reference  torus. 

b.  the  distance  between  the  centers  of  the  test  and  reference  torus 

c.  the  angle  between  the  planes  of  the  test  and  reference  torus 

Characteristics  of  Reliability 

Reliability  measures  the  size  of  the  class  of  problems  that  the  software  can  solve.  The 
objective  is  to  determine  the  sensitivity  of  CMS  software  to  variations  of  the  input  data.  Reflected 
in  the  input  data  are  variations  in  the  following  factors: 

FEATURE  GEOMETRY 

Notably  feature  size  and  location  may  affect  the  behavior  of  CMS  software.  Data  analysis 
software  may  be  less  reliable  for  large  size  or  features  located  far  from  the  origin. 

FEATURE  FORM  ERROR 

Feature  form  errors  (e.g.,  straightness,  roundness,  cylindricity,  etc.)  affect  the  software 
calculations  of  position,  size,  and  orientation. 

FEATURE  SAMPLING  STRATEGY 

The  number  of  points  sampled  and  the  pattern  in  which  the  points  are  taken  may  also  affect 
CMS  software  reliability.  Generally,  the  mathematical  minimum  number  of  points 
necessary  to  determine  a geometric  type  is  not  sufficient  for  the  measurement  of  an  actual 
feature. 
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POINT  MEASUREMENT  ERROR 

The  errors  in  each  sampled  point  induced  by  the  point  measurement  process  may  affect  the 
reliability  of  CMS  software. 
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Appendix  C: 


Special  Test  Report 


Template 


National  Institute  of  Standards  and  Technology 
Gaithersburg,  MD  20899-0001 


REPORT  OF  SPECIAL  TEST^ 


for 

« CMS  software  » 

Submitted  by: 

« customer  » 


This  software  package  was  tested  on  xxa  data  sets,  representing  the  following  geometry  types:  line, 
circle,  plane,  sphere,  cylinder,  cone,  and  torus,  and  following  the  test  procedures  documented  in 
NISTIR  xxxx.  The  results  of  the  tests  are  as  follows: 

Geometry  Type  Deviation  (pm)  Uncertainty  Ipml 
line 
circle 
plane 
sphere 
cylinder 
cone 
torus 


Computing  Environment:  Measurements  performed  by: 

Test  System  Version: 

Customer’s  Computing  Environment:  Cathleen  Diaz 

Special  Test  Date:  Computer  Scientist 

Report  Number: 

Approved  by: 

Neil  Christopher 
Group  Leader 

Page  1 of  3 1-301-975-3888 


^ This  report  gives  the  results  of  a single  special  test  conducted  by  the  Manufacturing  Systems  Integration  Division  of 
the  National  Institute  of  Standards  and  Technology.  Please  call  the  Manufacturing  Systems  Integration  Division  if  there 
are  any  questions  about  these  results  or  the  distinction  between  a special  test  and  calibration. 


« short  item  description  » 


The  reported  uncertainty  is  « explanation  of  where  uncertainty  came  from  ».  In  the  case  of 
special  tests,  NIST  will  generally  assign  an  estimated  “uncertainty”  to  the  result;  but  it  must  be 
recognized  by  the  user  that  such  uncertainty  figures  are  not  necessarily  supported  by  a long  term  data 
base. 

In  a rigorous  sense,  the  values  reported  in  this  Report  of  Special  Test  apply  to  the  software  tested 
only  in  the  computing  environment  in  which  it  was  tested.  NIST  can  provide  no  assurances  that  the 
customer’s  software  will  have  the  same  value  as  reported  by  NIST  when  used  in  the  customer’s 
facility  at  a later  date.  For  a characterization  of  errors  associated  with  this  test  see  reference  5. 

The  range  of  test  conditions  were  as  follows: 

Range  of  sampling  Range  of 

Geometry  Type  Range  of  form  plan  measurement  error 

errors 


line 

circle 

plane 

sphere 

cylinder 

cone 

torus 

This  Special  Test  was  carried  out  as  follows.  NIST  generated  data  sets  simulating  the  ranges  of  test 
conditions  described  above.  NIST  also  generated  reference  fit  results  using: 

□ NIST  Algorithm  Testing  System  Internal  Algorithms 

□ Other:  


The  customer  received  the  NIST-generated  data  sets  in  ASCII  format  and  generated  corresponding 
fit  results  using  the  software  under  test.  NIST  then  compared  each  of  the  customer’s  fits  to  the 
reference  fit  for  the  corresponding  data  set  using  procedures  set  forth  in  ASME  Standard  B89.4. 10. 
The  reported  test  results  for  each  geometry  type  are  the  maximum  observed  deviations  between  the 

Special  Test  Date: 

Report  Number: 

Page  2 of  3 


« short  item  description  » 


customer’s  fits  and  the  reference  fits  for  all  data  sets  corresponding  to  that  geometry  type. 
Detailed  data  concerning  this  Special  Test  are  available  from  NIST  on  request. 

« special  information  » 

For  detailed  descriptions  of  the  technical  approach  used  for  these  test  services  and  specifics  on  the 
test  procedures  see  the  following  references. 

[1]  ASME  B89.4. 10-  199x,  Draft,  Methods  for  Performance  Evaluation  of  Coordinate 
Measuring  System  Software,  American  Society  of  Mechanical  Engineers,  New  York, 
NY. 

[2]  Diaz,  C.,  NISTIR  5686,  1995,  Algorithm  Testing  and  Evaluation  Program  for 
Coordinate  Measuring  Systems:  Testing  Methods,  Gaithersburg,  MD.. 

[3]  Diaz,  C.,  NISTIR  5137,  1993,  User’s  Guide  for  the  Algorithm  Testing 
System/Version  1.1,  Gaithersburg,  MD. 

[4]  Diaz,  C.,  and  Hopp,  T.,  Evaluation  of  Software  for  Coordinate  Measuring  Systems, 
proceedings  of  the  1995  SME  Clinic,  CMMs  Week,  June  5-8,  Society  of 
Manufacturing  Engineers,  Dearborn,  MI;  proceedings  of  the  1995  Interface 
Symposium,  June  21-24,  Interface  Foundation  of  North  America,  Carnegie  Mellon 
University,  Pittsburgh,  PA. 

[5]  Hopp,  T.,  Performance  Measures  for  Geometric  Fitting  in  the  NIST  Algorithm 
Testing  and  Evaluation  Program  for  Coordinate  Measuring  Systems,  1995,  accepted 
for  publication  in  the  NIST  Journal  of  Research.  Gaithersburg,  MD. 

[6]  Rosenfeld,  D.,  NISTIR  5674,  1995,  User's  Guide  for  the  Algorithm  Testing  System 
Version  2,  Gaithersburg,  MD,. 

[7]  Rosenfeld,  D.,  NISTIR  xxxx.  Reference  Manual  for  the  Algorithm  Testing  System 
Version  2,  Gaithersburg,  MD,  in  preparation. 
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